Real time calibration for time-of-flight depth measurement

ABSTRACT

A method for determining a distance to a target object includes transmitting light pulses to illuminate the target object and sensing, in a first region of a light-sensitive pixel array, light provided from an optical feedback device that receives a portion of the transmitted light pulses. The feedback optical device includes a preset reference depth. The method includes calibrating time-of-flight (TOF) depth measurement reference information based on the sensed light in the first region of the pixel array. The method further includes sensing, in a second region of the light-sensitive pixel array, light reflected from the target object from the transmitted light pulses. The distance of the target object is determined based on the sensed reflected light and the calibrated TOF measurement reference information.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/721,640, filed on Sep. 29, 2017, entitled “REAL TIME CALIBRATION FORTIME-OF-FLIGHT DEPTH MEASUREMENT,” which is a non-provisional of andclaims the benefit of and priority to U.S. Provisional PatentApplication No. 62/402,770, filed on Sep. 30, 2016, entitled “REAL TIMECALIBRATION FOR TIME-OF-FLIGHT DEPTH MEASUREMENT” the content of whichare hereby incorporated by reference in their entirety for all purposes.

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to time-of-flight(TOF) depth measurement systems.

BACKGROUND OF THE INVENTION

A time-of-flight (TOF) camera is a range imaging camera system thatresolves distance based on the speed of light, measuring thetime-of-flight of a light signal between the camera and the subject foreach point of the image. With a time-of-flight camera, the entire sceneis captured with each laser or light pulse. Time-of-flight cameraproducts have become popular as the semiconductor devices became fast tosupport such applications. Direct Time-of-Flight imaging systems measurethe direct time-of-flight required for a single laser pulse to leave thecamera and reflect back onto the focal plane array. The 3D images cancapture complete spatial and temporal data, recording full 3D sceneswith a single laser pulse. This allows rapid acquisition and real-timeprocessing of scene information, leading to a wide range ofapplications. These applications include automotive applications,human-machine interfaces and gaming, measurement and machine vision,industrial and surveillance measurements, and robotics, etc.

The simplest version of a time-of-flight camera uses light pulses or asingle light pulse. The illumination is switched on for a short time,the resulting light pulse illuminates the scene and is reflected by theobjects in the field of view. The camera lens gathers the reflectedlight and images it onto the sensor or focal plane array. The time delaybetween the out-going light and the return light is the time-of-flight,which can be used with the speed of light to determine the distance. Amore sophisticated TOF depth measurement can be carried by illuminatingthe object or scene with light pulses using a sequence of temporalwindows and applying a convolution process to the optical signalreceived at the sensor.

SUMMARY OF THE INVENTION

Conventional time-of-flight depth measurement systems can be susceptibleto variations in processes, operating voltages, and thermal conditions.For example, the system might potentially be modified if the thermalconditions (T) or the operating voltages (V) of each of the componentswere modified. In addition, the TOF measurements can also be affected bythe frame rate of the camera. These variations might be dependent on theprocess (P) of each of the components. Although it might possible tostudy the nature of the PVT effect and prepare a model to compensate forthe errors, this process is time-consuming and may not necessarilyprovide a full coverage of the physical envelope. Further, a calibrationof full range depth measurement is undesirable, because it can take manyframes and increases operation overhead.

In order to mitigate the effects of variations in component processingor operating conditions, embodiments of the invention provide a systemand method for run-time calibration of TOF depth measurement using anoptical feedback device and a small dedicated feedback sensing region inthe sensor pixel array. The small number of feedback pixels allows forfast sensing and signal processing, and with the strong feedbackillumination, for example, provided by an optical fiber, the number ofsampling pulses can be greatly reduced. Many steps of illumination andreadout can be carried out in a short time for calibrating a wide rangeof depth. As a result, the depth calibration can be carried out over awide range of depth measurement at run-time in each frame withoutaffecting the frame rate of the camera. Isolation between the feedbackregion and active region of the pixel array is provided to minimizeinterference. Further, the overhead in power consumption or dedicatedfeedback pixels is limited.

In some embodiments of the present invention, a time-of-flight imagingsystem includes an illuminator to transmit light pulses to illuminate atarget object for determining a distance, or depth, to the targetobject. The imaging system has an image sensor with a light-sensitivepixel array to receive optical signals from the light pulses. The pixelarrays include an active region and a feedback region. An opticalfeedback device directs a portion of the light from the illuminator tothe feedback region of the pixel array. The optical feedback deviceprovides a preset reference depth for calibration. The imaging system isconfigured to transmit light pulses to illuminate a target object andsense, in the feedback region of the pixel array, light from the opticalfeedback device, using a sequence of shutter windows that includes delaytimes representing a range of depth. The imaging system is configured tocalibrate time-of-flight (TOF) depth measurement reference informationbased on the sensed light in the feedback region of the pixel array. ForTOF depth measurement, the imaging system is configured to sense, in theactive region of the light-sensitive pixel array, light reflected fromthe target object, and to determine the distance of the target objectbased on the sensed reflected light and the calibrated TOF measurementreference information.

In an embodiment of the above system, the TOF measurement referenceinformation includes a look-up table correlating distance of an objectto a ratio between two sampled light signals using two shutters withdifferent time delays.

In some embodiments of the present invention, in a digital cameracharacterized by a preset frame rate, a method is provided fordetermining a distance to the target object with distance calibration ina single frame period. The method includes transmitting light pulses toilluminate a target object and sensing, in a first region of alight-sensitive pixel array, light provided from an optical feedbackdevice that receives a portion of the transmitted light pulses. Thefeedback optical device includes a preset reference depth. The lightfrom the optical feedback device is sampled using a sequence of shutterwindows that includes delay times representing a range of distance. Themethod includes calibrating time-of-flight (TOF) depth measurementreference information based on the sensed light in the first region ofthe pixel array. The method further includes sensing, in a second regionof the light-sensitive pixel array, light reflected from the targetobject from the transmitted light pulses. The distance of the targetobject is determined based on the sensed reflected light and thecalibrated TOF measurement reference information.

In some embodiments of the present invention, a method for determining adistance to a target object includes transmitting light pulses toilluminate the target object and sensing, in a first region of alight-sensitive pixel array, light provided from an optical feedbackdevice that receives a portion of the transmitted light pulses. Thefeedback optical device includes a preset reference depth. The methodincludes calibrating time-of-flight (TOF) depth measurement referenceinformation based on the sensed light in the first region of the pixelarray. The method further includes sensing, in a second region of thelight-sensitive pixel array, light reflected from the target object fromthe transmitted light pulses. The distance of the target object isdetermined based on the sensed reflected light and the calibrated TOFmeasurement reference information.

The following description, together with the accompanying drawings, willprovide further understanding of the nature and advantages of theclaimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a time-of-flight (TOF) imaging systemfor depth measurement according to an embodiment of the presentinvention;

FIGS. 2A and 2B are diagrams illustrating examples of image sensor pixelarray in a time-of-flight (TOF) imaging system having fiber opticfeedback for calibration according to an embodiment of the presentinvention;

FIG. 3 is a diagram illustrating optical feedback paths in atime-of-flight (TOF) imaging system according to an embodiment of thepresent invention;

FIGS. 4A and 4B are diagrams illustrating a time-of-flight (TOF) imagingsystem with fiber optic feedback for calibration according to anembodiment of the present invention;

FIG. 5 is a timing diagram illustrating a method for time-of-flight(TOF) depth measurement according to an embodiment of the presentinvention;

FIG. 6 is a diagram illustrating sensed signal versus light to shutterdelay time according to an embodiment of the present invention;

FIG. 7A is a diagram illustrating sensed signals versus light to shutterdelay times of two signals with two shutters according to an embodimentof the present invention;

FIG. 7B is a diagram illustrating simulated signals versus light toshutter delay times of two signals with two shutters according to anembodiment of the present invention;

FIG. 7C is a diagram illustrating simulated signals versus depth for twosignals with two shutters according to an embodiment of the presentinvention;

FIG. 8 is a timing diagram illustrating a method for calibration anddepth measurement in a time-of-flight (TOF) imaging system according toan embodiment of the present invention;

FIG. 9 is another timing diagram illustrating a method for calibrationand depth measurement in a time-of-flight (TOF) imaging system accordingto an embodiment of the present invention;

FIG. 10 is a flowchart illustrating a method for calibration and depthmeasurement in a time-of-flight (TOF) imaging system according to anembodiment of the present invention;

FIG. 11 is a perspective view diagram illustrating a portion of atime-of-flight (TOF) imaging system with fiber optic feedback forcalibration according to an embodiment of the present invention;

FIG. 12 is a cross-sectional plan view of a portion of time-of-flight(TOF) imaging system of FIG. 11 according to an embodiment of thepresent invention;

FIG. 13 is a perspective view of a portion of time-of-flight (TOF)imaging system 1100 of FIG. 11 according to an embodiment of the presentinvention;

FIG. 14 is a cross-sectional plan view of a portion of time-of-flight(TOF) imaging system of FIG. 11 according to an embodiment of thepresent invention;

FIG. 15 is a cross-sectional side view of a portion of time-of-flight(TOF) imaging system of FIG. 11 according to an embodiment of thepresent invention;

FIG. 16 is another perspective view diagram illustrating a portion of atime-of-flight (TOF) imaging system with fiber optic feedback forcalibration according to an embodiment of the present invention;

FIG. 17 is a perspective view diagram illustrating the printed circuitboard (PCB) of the time-of-flight (TOF) imaging system with fiber opticfeedback for calibration according to an embodiment of the presentinvention;

FIG. 18 is a perspective view diagram illustrating a prism 1138 that canbe used in the time-of-flight (TOF) imaging system 1100 with fiber opticfeedback for calibration according to an embodiment of the presentinvention; and

FIG. 19 is a flowchart illustrating a method for forming atime-of-flight (TOF) imaging system with fiber optic feedback forcalibration according to an embodiment of the present invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Embodiments of the present invention provide a system and method thatenable TOF depth measurement with calibration to provide high accuracyusing optical feedback and fast image processing. A range of depthmeasurements can be calibrated for each frame with minimal effect onsensor performance and power consumption.

The description below is presented with reference to a series of drawingfigures enumerated above. These diagrams are merely examples, and shouldnot unduly limit the scope of the claims herein. In connection with thevarious aspects illustrated and described, one of ordinary skill in theart would recognize other variations, modifications, and alternatives.

FIG. 1 is a diagram illustrating a time-of-flight (TOF) imaging systemfor depth measurement according to an embodiment of the presentinvention. As shown in FIG. 1, a time-of-flight (TOF) imaging system100, also referred to as a TOF digital camera, includes an illuminator110 to transmit light pulses 112 to illuminate a target object 120 fordetermining a distance to the target object. Illuminator 110 can includea pulsed illumination unit and optics for emitting the light pulses 112toward the target object. In this example, illuminator 110 is configuredto transmit light to the target object using, for example, a laser lightsource. However, it is understood that other sources of electromagneticradiation can also be used, for example, infra-red light, radiofrequency EM waves, etc. Imaging system 100 also includes an imagesensor 130 having a gated sensor unit including a light-sensitive pixelarray to receive optical signals from the light pulses in the field ofview (FOV) 132 of the sensor lens. The pixel arrays including an activeregion and a feedback region, as explained below in connection withFIGS. 2A and 2B. Imaging system 100 also has an optical feedback device140 for directing a portion of the light from the illuminator 110 to thefeedback region of the pixel array. The optical feedback device 140provides a preset reference depth. The preset reference depth can be afixed TOF length, which can be used to produce a look up table (LUT)that correlates sensed light vs. depth measurement. In some embodiments,the optical feedback device can fold a direct light from theillumination unit into the field of view (FOV) of the lens in the sensorunit. Imaging system 100 further includes a TOF timing generator 150 forproviding light synchronization and shutter synchronization signals tothe illuminator and the image sensor.

In FIG. 1, TOF imaging system 100 is configured to transmit light pulsesto illuminate a target object 120. Imaging system 100 is also configuredto sense, in the feedback region of the pixel array, light from theoptical feedback device 140, using a sequence of shutter windows thatincludes delay times representing a range of depth. The range of depthcan include the entire range of distance that can be determined by theimaging system. Imaging system 100 calibrates time-of-flight (TOF) depthmeasurement reference information based on the sensed light in thefeedback region of the pixel array. Imaging system 100 is furtherconfigured to sense, in the active region of the light-sensitive pixelarray, light reflected from the target object, and to determine thedistance of the target object based on the sensed reflected light andthe calibrated TOF measurement reference information.

FIG. 2A is a simplified diagram illustrating a pixel array that can beused in imaging system 100 according to an embodiment of the presentinvention. As shown, pixel array 200 includes a plurality of pixels 212,and each pixel in the pixel array includes a photo sensitive element(e.g. a photo diode), which converts the incoming light into a current.Fast electronic switches are used as shutters to control the timing ofthe light sensing operation. A time-of-flight (TOF) camera acquiresdepth images by determining the time during which the light travels froma source to an object and to the sensor of the camera. This can be doneby illuminating the object or scene with light pulses using a sequenceof temporal windows and applying a convolution process to the opticalsignal received at the sensor. Further details are described below. Asshown in FIG. 2A, pixel array 200 includes an active region 210 and afeedback region 220. The active region can be used for determining thedistance of a target object, and the feedback region can be used fordepth calibration. The pixel array can also include an isolation region221 separating the feedback region 220 from the active region 210 toreduce interference. The dimension of the isolation region can beselected to prevent the light from the feedback loop to contaminate theimaging signal collected by the objective lens. In some embodiments, forexample, the isolation region can have a width of about 100 μm-200 μm.In some embodiments, feedback region 220 can be located in part of thepixel array that is outside the field of view, e. g. in a corner, or ina less used region of the pixel array. Therefore, the dedicated feedbackregion of the sensor does not incur much overhead. The small feedbackregion can have a limited number of pixels, for example, from a singlepixel to a 10×10 array of pixels, which allows for fast sensing andsignal processing. In some embodiments, a larger feedback region can beused to provide better signal-to-noise ratio (SNR). Averaging the pixelsin a small array can contribute to the accuracy. In some embodiments,both the feedback and active regions are exposed during the calibrationphase, separately. The difference between the two can be used for thecompensation at run time.

FIG. 2B is a simplified diagram illustrating a pixel array that can beused in imaging system 100 according to another embodiment of thepresent invention. As shown in FIG. 2B, pixel array 250 is similar topixel array 200 of FIG. 2A, but can have more than one feedback regions.Pixel array 250 includes an active region 210 and two or more feedbackregions 220. The pixel array can also include an isolation region 221separating each feedback region from the active region. The isolationregion can reduce interference between the feedback region and theactive region. Pixel array 250 can be used in a TOF imaging systemhaving two illumination sources. In some embodiments, an imaging systemcan include more than two illumination sources and correspondingfeedback sensor regions.

FIG. 3 is a simplified schematic diagram illustrating a portion of thetime-of-flight (TOF) imaging system 100 of FIG. 1. FIG. 3 illustratesthat the optical feedback device is configured to prevent light leakagefrom the optical feedback device 140 to the normal pixels in the array.Light inserted in the edge of the FOV can only hit specific pixels inthe pixel array and light having different angle cannot enter the opticsof the sensor.

In some embodiments, the optical feedback device can be configured tofold a direct light from the illumination unit into the field of view(FOV) of the lens in the sensor unit. FIGS. 4A and 4B are simplifieddiagrams illustrating a time-of-flight (TOF) imaging system 400 withfiber optic feedback for calibration according to an embodiment of thepresent invention. FIG. 4A is a top view and FIG. 4B is a sidecross-sectional view of the imaging system. Imaging system 400 includesan illumination unit 410 and a sensor unit 430 disposed on a printedcircuit board (PCB) 401. As shown in FIGS. 4A and 4B, illumination unit410 includes a diode laser source 412, a collimating lens 414, and adiffuser 416 inside an illumination housing 418. Sensor unit 430includes an image sensor 432, a lens 434, and a lens barrel 436 that ismounted on the image sensor with an adhesive 438. Imaging system 400also has an optical fiber 420 to provide the feedback path. In thisembodiment, optical fiber 420 collects certain amount of light from theinterior of the illumination housing (e. g., from parasitic reflectionsinside) and directs it to a corner 442 of a pixel array 440 of imagesensor 432, but outside lens barrel 436. In some embodiments, opaqueadhesive 438 blocks the light from entering the lens barrel. In thisexample, corner region 442 of the pixel array serves as the feedbackregion of the image sensor.

FIG. 5 is a timing diagram illustrating a method for time-of-flight(TOF) depth measurement according to an embodiment of the presentinvention. In FIG. 5, the horizontal axis is the time, and the verticalaxis is the intensity or magnitude of the light signal. Waveform 1represents the light pulse arriving at the sensor, which can bereflected from the target or provided by the feedback optical device.Wave form 2 represents the shutter window. It can be seen that the lightpulse has a width W_(light), and the shutter window has a width ofW_(shutter). Further, there is a time delay between the leading edge ofthe light and the shutter, D_(L->SH). It can be seen that the amount oflight sensed by the sensor varies with the relative delay of the shutterwith respect to the light.

FIG. 6 is a diagram illustrating the magnitude of sensed light signalversus light-to-shutter delay time according to some embodiments of thepresent invention. In FIG. 6, the horizontal axis is thelight-to-shutter delay, D_(L->SH), and the vertical axis is the amountof light sensed by the sensor. The diagram is divided into severalregions, 601 to 605, In region 601, the shutter window is far ahead ofthe light pulse (to the left) and the shutter is already closed beforethe light arrives. In other words, the light-to-shutter delay isnegative. Thus, there is no overlap between the shutter and the light.The delay increases moving to the right of the horizontal axis. At point611, the shutter starts to overlap with the light. As the delayincreases further through region 602, the overlap between the shutterand the light continues to increase, and more light is sensed, resultingin the rising curve in region 602. At point 612, the full width of thelight starts to overlap with the shutter window. In region 603, theshutter is fully open throughout the duration of the light pulse, andthe width of region 603 is determined by the width of the shutteropening W_(shutter) minus the width of the light pulse W_(light). Themagnitude of light received in this region is marked “Shutter ONsignal.” At point 613, the rising edge of the shutter window is alignedwith the rising edge of the light pulse, and the delay D_(L->SH) iszero, as marked by point 617. In region 604, the delay D_(L->SH)continues to increase, and the overlap between the shutter window andthe light decreases. As a result, the magnitude of the sensed lightdecreases in this region, as shown by the declining curve. At point 615,the delay is equal to the light width, and the shutter opens as thelight pulse ends; as a result, no light is sensed. In region 605, theshutter opens after the light pulse has already passed. No light issensed in region 605, and the amount of sensed light in this region ismarked “Shutter OFF signal.” Note that in regions 602 and 604, theamount of light collected by the sensor varies depending on thelight-to-shutter delay, D_(L->SH). These regions are used in TOF depthmeasurement calibration, as explained below.

FIG. 7A is a diagram illustrating sensed light signals versuslight-to-shutter delay times of two signals with two shutters accordingto an embodiment of the present invention. The time-of-flight (TOF)camera acquires depth images by determining the time which light needsfrom a source to an object and reflected back to the camera. This is canbe done by illuminating the object or scene with a light pulse andapplying a convolution of a sequence of windows with varying delay timesto the received optical signal by the sensor. In some embodiments,multiple groups of calibration light pulses are transmitted using asequence of shutter windows that includes delay times representing arange of depth. Each group of light pulses is followed by a readoutoperation. In each readout, the light from the optical feedback deviceare sensed in the feedback region of the pixel array of the sensor. Thereadout data is then analyzed using a convolution process to determineTOF depth data. As described above, in regions 602 and 604 of FIG. 6,the amount of light collected at the sensor varies depending on thelight-to-shutter delay, D_(L->SH). Sensed light data similar to that inFIG. 6 can be collected. These regions are used in TOF depth measurementcalibration. As shown in FIG. 7A, two calibration sequences can becarried out to reduce the effect of unknown reflectivity of the targetobject; and the two sequences are denoted S1 and S2. In an embodiment,the difference in light-to-shutter delay, D_(L->SH) for the twosequences is equal to the width of the shutter window W_(shutter). Underthis condition, region 604 of sequence S1 and region 602 of sequence S2can be aligned in the plot of FIG. 7A and form slices t-1, t-2, ... ,t-k. In each slice, the amount of light collected in S1 and S2,respectively, represent two portions of the reflected light pulse, andthe ratio of S2/S1 is related to a corresponding depth or distance tothe target object. The region between points A and B in FIG. 7Arepresents the depth range that can be determined by this TOF imager.Data of received light can be collected by measuring at multiple pointswith delays between A and B in front of a target. Using a convolutionprocess, a look up table (LUT) can be constructed that relates the ratioS2/S1 to the depth or distance to the target. The initial lookup tablecan be constructed in the factory calibration process. In a subsequentTOF depth measurement, two measurements are made with delays from thesame slice of time in FIG. 7A. A ratio of sensed light S2/S1 isdetermined based on sensed data, and the corresponding depth can bedetermined from the look up table.

FIG. 7B is a diagram illustrating simulated signals versuslight-to-shutter delay times of two signals with two shutters accordingto an embodiment of the present invention. The simulation was carriedout with two shutters on a static test with a flat target at 100 cm fromthe camera, scanning a range of the light-to-shutter delays. Similar toFIG. 7A, the shutter signal (or the number of photo-electrons collectedat the sensor) is plotted for two shutters S1 and S2. In this view,depth can be negative. In the horizontal axis of FIG. 7B, the delay isconverted into depth by the following equation:

<depth>=<speed of light>/2* (<electronic delay>−<simulation delayvector>)

In some embodiments, the width of the light pulse is 5-10 nsec, and theshutter window width is 5-15 nsec. The range of delays examined isbetween 5-20 nsec. In some embodiments, the light pulse width can bebetween 3 nsec to 20 sec. The width of the shutter can be in the samerange.

FIG. 7C is a diagram illustrating simulated signals versus depth for twosignals with two shutters according to an embodiment of the presentinvention. FIG. 7C shows data as measured on a rail against a wall atdifferent distances (with 1/distance² decay). It can be seen that thereis a correlation between the ratio of S2/S1 and the depth.

From testing data such as those obtained using methods described inFIGS. 5, 6, and 7A-7C, a look up table (LUT) is constructed in thefactory calibration process. In a TOF depth measurement, a ratio ofS2/S1 is determined based on sensed data, and the corresponding depthcan be determined from the lookup table.

As described above, time-of-flight depth measurement systems can besusceptible to variations in process and operating conditions, such astemperature, voltage, and frame rate, etc. In order to mitigate theeffects of variations, embodiments of the invention provide a system andmethod for run-time calibration of TOF depth measurement using anoptical feedback device as described above. The small number of feedbackpixels allows for fast sensing and signal processing, and with thestrong feedback illumination, for example provided by optical fiber, thenumber of sampling pulses can be greatly reduced. The process ofillumination and readout can be carried out in a short time. As aresult, the depth calibration can be carried out at run-time withoutaffecting the frame rate of the camera. The calibration can be carriedout in each frame. Further, the overhead in power consumption ordedicated feedback pixels is small. Isolation between the feedbackregion and active region of the pixel array is provided to minimizeinterference.

FIG. 8 is a timing diagram illustrating a method for depth profilecalibration between frames of time-of-flight depth measurement accordingto an embodiment of the present invention. The method includes astabilization period 810, a calibration period 820, and a measurementperiod 830. In stabilization period 810, thermal stabilizationillumination pulses are emitted, followed by a dummy readout for thermalstabilization of the sensor. In calibration period 820, thetime-of-flight lookup table (LUT) is calibrated. Here, multiple groupsof calibration illumination pulses P-1, P-2, . . . , P-N are emittedusing a sequence of shutter windows that includes delay timesrepresenting a range of depth. Each group of light pulses is followed bya readout operation, R-1, R-2, . . . , R-N, respectively. In eachreadout, the light from the optical feedback device are sensed in thefeedback region of the pixel array of the sensor. The readout data isthen analyzed using a convolution process to determine TOF depth data asdescribe above in connections with FIGS. 5, 6, and 7A-7C. The depth datais then used to calibrate the lookup table.

The measurement period 830 has two steps 831 and 832. In the first step831, a first group of light pulses S1 with a first shutter delay D1 istransmitted to illuminate the target. Because only a small amount oflight can be collected by the sensor within a shutter window, often alarge number, e.g., several thousand, pulses are sent out and gatheredto increase the signal to noise ratio. During the “S1 read” period, thelight reflected from the target is sensed in the active region of thepixels in the sensor. In the second step 832, a second group of lightpulses S2 with a second shutter delay D2 is transmitted to illuminatethe target. During S2 read, the light reflected from the target issensed in the active region of the pixels in the sensor. Next, the ratioof sensed data readouts S2/S1 are used to determine the distance of thetarget object using the calibrated look up table. In some embodiments,S1 and S2 have preset delays that are chosen in the factory calibrationprocess or in the field of application.

FIG. 9 is a timing diagram illustrating that the depth profilecalibration can fit in between frames of time-of-flight depthmeasurement according to an embodiment of the present invention. FIG. 9is similar to FIG. 8, and further includes examples of the length oftime each operation takes within a frame of time-of-flight depthmeasurement. In this embodiment, the thermal stabilization pulses take0.15 msec, and the dummy readout for thermal stabilization takes 0.1msec. Therefore, the length of the stabilization period is about 0.25msec. In the look up table (LUT) calibration period 820, 20 steps ofcalibration light pulses and readouts are used, each with a differentlight-to-shutter delay time. In an example, each step includes 30pulses, each having a pulse width of 150 nsec, followed by a read outoperation of 3 μsec. Thus, the calibration period takes about 0.15 msec.In the measurement period 830, the S1 step can include 1.5 msec of lightpulses (e.g., 1,000 pulses of 150 nsec pulses) followed by a 0.5 msecreadout. Similarly, the S2 step can include 2.0 msec of light pulses,followed by a 0.5 msec readout. In this example, the complete operationincluding stabilization, full range depth calibration, and TOF depthmeasurement takes 4.9 msec. The calibration phase takes about 1/300 ofthe total operation. This optical operation is fast enough to fit in aframe rate of 60 or more frames per second (fps).

The embodiments of the invention provide many advantages overconventional methods. For example, the feedback optical device canprovide strong light for calibration. For example, the feedback opticaldevice can include optical fiber. One or more separate feedback regionsin the pixel array are used for sensing the feedback optical signal. Thefeedback regions are configured in unused or less used regions of thepixel array, and is much smaller than the active region of the array.For example, several pixels are sufficient for feedback sensing if thefeedback optical device can provide a strong signal. The small feedbacksensing region enables quick sensing and fast processing of sensed data,allowing fast calibration of the depth range of interest.

FIG. 10 is a simplified flowchart illustrating a method for TOF depthmeasurement including full range depth calibration according to anembodiment of the present invention. The method described above can besummarized in the flowchart of FIG. 10. As shown, method 1000 includestransmitting light pulses to illuminate a target object, at step 1010.Next, at step 1020, light provided from an optical feedback device issensed in a first region of a light-sensitive pixel array. Here, thefirst region is used as the feedback region. The optical feedback devicereceives a portion of the transmitted light pulses. The feedback opticaldevice includes a preset reference depth for TOF depth measure. Thelight from the optical feedback device is sampled using a sequence ofshutter windows that includes delay times representing a range ofdistances. For TOF depth measurement, the method includes sensing, in asecond region of the light-sensitive pixel array, the scene data whichis light reflected from the target object from the transmitted lightpulses, at step 1030. The second region is the active region of thepixel array. The method includes calibrating time-of-flight (TOF) depthmeasurement reference information based on the sensed light in the firstregion of the pixel array, at step 1040. This process is described indetails above in connection with FIGS. 5, 6 and 7A-7C. Note that,depending on the embodiments, steps 1030 and 1040 can be carried out inany order. For example, after the calibration data (1020) and the scenedata (1040) are captured, data calibration can be processed first andthen the scene data is processed. Alternative, both TOF data calibrationand scene data processing can be carried out simultaneously. Next, themethod includes, at step 1050, determining a distance of the targetobject based on the sensed reflected light and the calibrated TOFmeasurement reference information.

In some embodiments, the method can be carried out in a digital cameracharacterized by a preset frame rate. The calibration can fit in asingle frame period of the camera. In an embodiment, the light from theoptical feedback device is sampled using a sequence of shutter windowsthat includes delay times representing a range of distance. Aconvolution process is then used to correlate the measured signals withthe distance.

Embodiments of the invention provide integrated mechanics, electronics,and optics design and processing procedures to form a miniature circuitfor optical feedback for high speed time-of-flight (TOF) signal from theillumination portion into the corner pixels within the image sensor. Thedesign includes high level of integration, efficient transfer of lightin the feedback loop to save energy consumption, and robustness overproduction tolerances. The embodiments can also avoid occlusion in theimaging optics, not add constraints on the back focal length of theobjective lens, and avoid leakage of light from the feedback loop intothe imaging area, and vice versa. Further details are described below.

FIG. 11 is a perspective view diagram illustrating a portion oftime-of-flight (TOF) imaging system 1100 with fiber optic feedback forcalibration according to an embodiment of the present invention. Imagingsystem 1100 includes an illumination unit 1110 and a sensor unit 1130disposed on a printed circuit board (PCB) 1101. As shown in FIG. 11,illumination unit 1110 includes a laser diode source 1112, anillumination optics 1114, and a diffuser 1116. The illumination optics1114 is disposed in a cavity 1118 in the PCB 1101, and can includeprisms with integrated lenses to fold and collimate the light beam fromthe laser diode. Sensor unit 1130 includes an image sensor 1132 with apixel array 1136, and an objective lens 1134 (shown in broken lines)mounted on the image sensor 1132. As shown in FIG. 11, for TOF imaging,emitted light 1111 from diffuser 1116 is directed to a target objectdisposed above and outside the drawings in FIG. 11, and light 1131reflected from the target object enters through objective lens 1134 toreach the pixel array in image sensor 1132. Imaging system 1100 also hasan optical fiber 1120 to provide the feedback path. In this embodiment,an optical fiber 1120 collects a small amount of light from the interiorof the illumination optics housing (e. g., from parasitic reflectionsinside) and directs it to a prism 1138 that is attached to a corner 1142of the pixel array of image sensor 1132. In this example, corner region1142 of the pixel array serves as the feedback region of the imagesensor.

FIG. 12 is a cross-sectional plan view of a portion of time-of-flight(TOF) imaging system 1100 of FIG. 11 according to an embodiment of thepresent invention. A laser diode 1112 is coupled to the base of thehousing of the illumination unit. An optical fiber 1120 collects a smallamount of light from the interior of the illumination optics housing (e.g., from parasitic reflections inside) and directs it to a prism 1138that is attached to a corner 1142 of the pixel array of image sensor1132. In some embodiments, no specific optical element in theillumination unit housing is needed to direct a portion of the laserlight to the optical fiber 1120. Optical fiber 1120 directs the light toprism 1138 attached to a sample area 1142 of the pixel array. In thisembodiment, scattered light from the illumination housing is collectedby the optical fiber 1120, and scattered light from the prism 1138 canbe sensed in the feedback sense area 1142 of the pixel array.

In FIGS. 11 and 12, optical fiber 1120 is coupled to the hypotenuse ofthe prism 1138. However, the coupling can also be made to a short side(cathetus) of the prism, as shown in FIGS. 13 and 14.

FIG. 13 is a perspective view of a portion of time-of-flight (TOF)imaging system 1100 of FIG. 11 according to an embodiment of the presentinvention. A laser diode 1112 is coupled to the base of the housing ofthe illumination unit. An optical fiber 1120 collects a small amount oflight from the interior of the illumination optics housing (e. g., fromparasitic reflections inside) and directs it to a prism 1138 that isattached to a corner 1142 of the pixel array 1136 of image sensor 1132.In this embodiment, scattered light from the illumination housing iscollected by the optical fiber 1120. Optical fiber 1120 directs thescattered light to prism 1138, which is attached to a sample area 1142of the pixel array 1136. Light from the prism 1138 can be sensed in thefeedback sense area 1142 of the pixel array 1136 for calibration.

FIG. 14 is a cross-sectional plan view of a portion of time-of-flight(TOF) imaging system 1100 of FIG. 11 according to an embodiment of thepresent invention. A laser diode 1112 is coupled to the base of thehousing of the illumination unit. An optical fiber 1120 collects a smallamount of scattered light from the interior of the illumination opticshousing (e. g., from parasitic reflections inside) and directs it to aprism 1138 that is attached to a corner 1142 of the pixel array of imagesensor 1132. Optical fiber 1120 directs the light to prism 1138 attachedto a sample area 1142 of the pixel array. Light from the prism 1138 canbe sensed in the feedback sense area 1142 of the pixel array 1136.

FIG. 15 is a cross-sectional side view of a portion of time-of-flight(TOF) imaging system 1100 of FIG. 11 according to an embodiment of thepresent invention. Imaging system 1100 in FIG. 15 includes anillumination unit 1110 and a sensor unit 1130 disposed on a printedcircuit board (PCB) 1101. As shown in FIG. 15, illumination unit 1110includes a laser diode source (not shown), an illumination optics 1114.Sensor unit 1130 includes an image sensor 1132 with a pixel array 1136,and an objective lens 1134 mounted on the image sensor 1132. As shown inFIG. 15, for TOF imaging, emitted light 1111 from illumination unit 1110is directed to a target object disposed above and outside the drawingsin FIG. 15, and light 1131 reflected from the target object entersthrough objective lens 1134 to reach the pixel array 1136 in imagesensor 1132. Imaging system 1100 also has an optical fiber 1120 toprovide the feedback path. In this embodiment, optical fiber 1120collects a small amount of scattered light from the interior of theillumination optics housing and directs it to a prism 1138 that isattached to a corner 1142 of the pixel array 1136 of image sensor 1132.In this example, corner region 1142 of the pixel array serves as thefeedback region of the image sensor.

FIG. 16 is another perspective view diagram illustrating a portion oftime-of-flight (TOF) imaging system 1100 with fiber optic feedback forcalibration according to an embodiment of the present invention. Asshown in FIG. 16, illumination unit 1110 includes a laser diode source1112, an illumination optics 1114, and a diffuser 1116. The illuminationoptics 1114 is disposed in a cavity 1118 in the PCB 1101, and caninclude prisms with integrated lenses to fold and collimate the lightbeam from the laser diode. Sensor unit 1130 includes an image sensorwith a pixel array 1136, and an objective lens 1134 (shown in brokenlines) mounted on the image sensor 1132. As shown in FIG. 16, for TOFimaging, emitted light 1111 from diffuser 1116 is directed to a targetobject disposed above and outside the drawings in FIG. 11, and light1131 reflected from the target object enters through objective lens 1134to reach the pixel array 1136 in the image sensor. The Imaging systemalso has an optical fiber 1120 to provide a feedback path. In thisembodiment, optical fiber 1120 collects a small amount of light from theinterior of the illumination optics housing (e. g., from parasiticreflections inside) and directs it to a prism 1138 that is attached to acorner 1142 of the pixel array of image sensor 1132. In this example,corner region 1142 of the pixel array serves as the feedback region ofthe image sensor.

In FIG. 16, light beam A goes from the laser into a folding surface ofthe prism in the illumination optics 1114. Light beam A1 reflected backfrom the prism entrance as stray light into a cavity in the illuminationunit. Light beam B exits collimated from the prism and is directed tothe diffuser 1115. Light beam B1 is reflected back from the diffuser asstray light into the cavity. The stray light from the cavity iscollected by the optical fiber 1120. In embodiment of the invention, theamount of stray light was measured by using a fiber connect to opticalpower meter to verify that enough light signal can be collected forfeedback calibration.

FIG. 17 is a perspective view diagram illustrating the printed circuitboard (PCB) 1101 of the time-of-flight (TOF) imaging system 1100 withfiber optic feedback for calibration according to an embodiment of thepresent invention. As shown in FIG. 17, PCB 1101 includes a first cavity1118 for the illumination unit, a second cavity 1139 for the imagesensor, and a third cavity 1121 for the optical fiber. FIG. 17 alsoillustrates a laser diode 1112 disposed on PCB 1101.

FIG. 18 is a perspective view diagram illustrating a prism 1138 that canbe used in the time-of-flight (TOF) imaging system 1100 with fiber opticfeedback for calibration according to an embodiment of the presentinvention. In some embodiments, the prism can be made of glass, or othersuitable transparent optical material. The prism has a top surface and abottom surface, both labeled as surface “1,” and three side surfaceseach labeled as surface “2.” As illustrated in FIGS. 11-16, the lightfrom the optical fiber enters the prism through one of the side surfaces2 and exits through the bottom surface 1 to be coupled into the pixelarray in the image sensor. Therefore, in some embodiments, the topsurface and two of the side surfaces of the prism are coated withreflective coating. The bottom surface and one of the side surfaces areuncoated and remains clear to allow the light from the optical fiber toenter the prism through a side surface and exit the prism through thebottom surface. In some embodiments, the prism is glued with atransparent adhesive to the pixel array and then coated with an opaquepaste to isolate the feedback optics from the imaging optics.

In some embodiments, such as those illustrated in FIGS. 11, 12, and 16,optical fiber 1120 is coupled to the hypotenuse of the prism 1138, and ashort side of the prism is disposed to overlap with the pixel array. Asimulation study shows that in these embodiments, about 5.8% of thelight coming through the fiber reaches the pixels area. In otherembodiments, such as those illustrated in FIGS. 13 and 14, optical fiber1120 is coupled to a short side of the prism 1138, and the hypotenuse ofthe prism is disposed to overlap with the pixel array. A simulationstudy shows that in these embodiments, about 8.7% of the light comingthrough the fiber reaches the pixels area.

FIG. 19 is a flowchart illustrating a method for forming atime-of-flight (TOF) imaging system with fiber optic feedback forcalibration according to an embodiment of the present invention. Inprocess 1910, the method includes forming the PCB assembly. The methodincludes forming the image sensor assembly by chip-on-board (CoB), inprocess 1920. In process 1930, the laser diode assembly is formed bychip-on-board (CoB). The method includes dispense clear adhesive onimage sensor for the optical feedback (OFB) prism in process 1940. TheOFB prism refers to optics that folds a direct light from theillumination unit into the field of view (FOV) of the lens. Next, theprism is attached to the PCB, in process 1950, and the adhesive is curedusing a UV cure process (1960). The method also includes attachingoptical fiber and dispensing clear adhesive during the attach time, inprocess 1970, and performing UV cure in process 1980. Further, inprocess 1990, the method includes dispensing absorbing adhesive materialon the prism and on the clear adhesive.

In some embodiments, the optical fiber can be a 250 optical gradeunjacketed plastic optical fiber. Other suitable optical fiber can alsobe used in alternative embodiments. The adhesive material mentionedabove can be a precision positioning optical adhesive or a suitablealternative adhesive material.

While the preferred embodiments of the invention have been illustratedand described, it will be clear that the invention is not limited tothese embodiments only. Numerous modifications, changes, variations,substitutions and equivalents will be apparent to those skilled in theart without departing from the spirit and scope of the invention asdescribed in the claims.

1. (canceled)
 2. A time-of-flight (TOF) imaging system, comprising: anilluminator to transmit light pulses to illuminate a target object fordetermining a distance to the target object; an image sensor having alight-sensitive pixel array to receive optical signals from the lightpulses, the pixel array including an active region and a feedbackregion; and an optical feedback device for directing a portion of thelight from the illuminator to the feedback region of the pixel array,the optical feedback device including a preset reference depth; whereinthe imaging system is configured to: in a calibration period, transmit agroup of calibration light pulses to illuminate the target object;sense, in the feedback region of the pixel array, light from the opticalfeedback device, using a sequence of calibration shutter windowscharacterized by delay times representing a range of depth; andcalibrate TOF depth measurement reference information based on thesensed light in the feedback region of the pixel array; in a measurementperiod, transmit a first measurement light pulse to illuminate thetarget object; sense, in the active region of the light-sensitive pixelarray, light reflected from the target object, using a first measurementshutter window with a first measurement delay time relative to the firstmeasurement light pulse; transmit a second measurement light pulse toilluminate the target object; sense, in the active region of thelight-sensitive pixel array, light reflected from the target object,using a second measurement shutter window with a second measurementdelay time relative to the second measurement light pulse; and determinethe distance of the target object based on light sensed using the firstmeasurement shutter window and the second measurement shutter window andthe calibrated TOF depth measurement reference information.
 3. Thesystem of claim 2, wherein the sequence of calibration shutter windowscomprises a first series of calibration windows and a second series ofcalibration windows, and delay times associated with the second seriesof calibration windows are offset from delay times associated with thefirst series of calibration windows by a width of the shutter window. 4.The system of claim 3, wherein calibrating time-of-flight depthmeasurement reference information comprises associating the range ofdepth with ratios of light signals sensed using the first series ofcalibration windows and the second series of calibration windows.
 5. Thesystem of any of claims 3, wherein the second measurement delay time isoffset from the first measurement delay time by the width of the shutterwindow.
 6. The system of any of claims 4, wherein determining thedistance of the target object comprises comparing a ratio of sensedlight signals using the first measurement delay time and the secondmeasurement delay time with the time-of-flight depth measurementreference information.
 7. The system of any of claims 3, wherein theoptical feedback device comprises an optical fiber configured to couplelight from inside an illuminator housing to the feedback region at acorner of the pixel array.
 8. The system of any of claims 6, wherein theTOF measurement reference information comprises a look-up tablecorrelating a distance of an object to a ratio between two sampled lightsignals sensed using the first series of calibration windows and thesecond series of calibration windows.
 9. The system of any of claims 2,wherein the calibration period and the measurement period are includedin a single frame period.
 10. The system of any of claims 2, whereincalibration takes about 1/300 of a frame period.
 11. The system of anyof claims 2, further comprising multiple illumination sources andcorresponding optical feedback regions.
 12. The system of any of claims2, wherein the imaging system is configured to calibrate time-of-flight,TOF, depth measurement reference information after sensing, in theactive region of the light-sensitive pixel array, light reflected fromthe target object.
 13. A method for calibrating a time-of-flight (TOF)camera system, comprising: transmitting a group of calibration lightpulses to illuminate a target object; sensing, in a first region of alight-sensitive pixel array, light provided from an optical feedbackdevice that receives a portion of the group of calibration light pulses,using a sequence of calibration shutter windows characterized by delaytimes relative to the calibration light pulses representing a range ofdepth, the optical feedback device characterized by a preset referencedepth; calibrating TOF depth measurement information based on the lightprovided from the feedback optic device; transmitting a firstmeasurement light pulse to illuminate the target object; sensing, in asecond region of the light-sensitive pixel array, light reflected fromthe target object, using a first measurement shutter window with a firstmeasurement delay time relative to the first measurement light pulse;transmitting a second measurement light pulse to illuminate the targetobject; sensing, in the second region of the light-sensitive pixelarray, light reflected from the target object, using a secondmeasurement shutter window with a second measurement delay time relativeto the second measurement light pulse; and determining a distance of thetarget object based on light sensed using the first measurement shutterwindow and the second measurement shutter window and the calibrated TOFdepth measurement information.
 14. The method of claim 13, wherein thesequence of calibration shutter windows comprises a first series ofcalibration windows and a second series of calibration window, and delaytimes associated with the second series of calibration windows areoffset from delay times associated with the first series of calibrationwindows by a width of the shutter window.
 15. The method of claim 14,wherein calibrating TOF depth measurement reference informationcomprises associating the range of depth with ratios of light signalssensed using the first series of calibration windows and the secondseries of calibration windows.
 16. The method of any of claims 15,wherein the TOF depth measurement reference information comprises alook-up table correlating a distance of an object to a ratio between twosampled light signals using the first series of calibration windows andthe second series of calibration windows.
 17. In a digital cameracharacterized by a preset frame rate, a method, comprising: in a singleframe period determined by the preset frame rate, transmittingcalibration light pulses to illuminate a target object; sensing, in afirst region of a light-sensitive pixel array, light provided from anoptical feedback device that receives a portion of the transmitted lightpulses, the feedback optical device including a preset reference depth,wherein the light from the optical feedback device is sampled using asequence of shutter windows that includes delay times relative to thetransmitted calibration light pulses representing a range of distance;calibrating time-of-flight (TOF) depth measurement reference informationbased on the sensed light in the first region of the pixel array;transmitting measurement light pulses to illuminate a target object;sensing, in a second region of the light-sensitive pixel array, lightreflected from the target object from the transmitted light pulses usinga measurement shutter windows with delay times relative to themeasurement light pulses; and determining a distance of the targetobject based on the sensed light reflected from the target object andthe calibrated TOF measurement reference information.
 18. The method ofclaim 17, wherein the optical feedback device comprises an opticalfiber.
 19. The method of claim 18, wherein the optical fiber isconfigured to couple light from inside an illuminator housing to thefirst region at a corner of the pixel array.
 20. The method of claim 17,wherein, wherein the TOF measurement reference information comprises alook-up table correlating distance of an object to a ratio between twosampled light signals using two shutters with different time delays. 21.The method of claim 17, wherein sensing light reflected from the targetobject comprises sensing light using two shutters with different timedelays with respect.